Concerning the high costs of traditional action recognition algorithm in color video and poor recognition performance caused by insufficient two-dimensional information, a new human action recognition method based on three-dimensional depth image sequence was put forward. On the temporal dimension, Temporal Depth Model (TDM) was proposed to describe the action. Specially, the entire depth maps were divided into several sub-actions under three orthogonal Cartesian planes. The absolute difference between two consecutive projected maps was accumulated to form a depth motion map to describe the dynamic feature of an action. On the spatial-dimension, Spatial Pyramid Histogram of Oriented Gradient (SPHOG) was computed from the TDM for the representation of an action to obtain the final descriptor. Support Vector Machine (SVM) was used to classify the proposed descriptors at last. The proposed method was tested on two authoritative datasets including MSR Action3D dataset and MSRGesture3D dataset, the recognition rates were 94.90% (cross subject test) and 94.86% respectively. The experimental results demonstrate that the proposed method has fast speed and better recognition, also it meets the real-time requirement in the depth video sequence system basically.